Predicting Deeper into the Future of Semantic Segmentation — Supplementary Material —

Let us consider a spatial position p of the frame we wish to predict Xt+1. The original baseline sets X̂t+1(p) by bilinearly interpolating the values of Xt surrounding spatial position p+d, where d = Ft→t−1(p). The issue is that one should be using the flow vector d′ = Ft+1→t(p) instead, which we cannot access since we are trying to predict Xt+1. The displacements d and d′ are in general not equal, since the physical point corresponding to d may have been replaced by another, which potentially does not have the same displacement. For instance, this is the case for still points that are about to be occluded by moving objects. In this case, d is zero, whereas d′ could correspond to a fast moving point. A qualitative example is shown is Figure 1, where the values for the pixels in front of the moving car are not replaced by those of the car as they should be. Note that this is a systematic failure case of the flow baseline used in [4] and [5], which concerns all pixels which are about to be occluded by a moving object. We call this baseline “t + 1 centric”, as it can be viewed as looping over the spatial positions of the prediction X̂t+1.